Skip to content

evetest: port Eden app and device telemetry smoke tests - #6277

Merged
milan-zededa merged 3 commits into
lf-edge:masterfrom
rene:evetest-app-tests
Aug 6, 2026
Merged

evetest: port Eden app and device telemetry smoke tests#6277
milan-zededa merged 3 commits into
lf-edge:masterfrom
rene:evetest-app-tests

Conversation

@rene

@rene rene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Description

Ports six Eden smoke tests to the evetest framework, in two new test packages
plus one new test app:

evetest/tests/apps/ (TestApplicationSuite) — from tests/eclient/testdata/:

  • TestAppInstanceMetadata (metadata.txt) — app POSTs to the link-local
    metadata server; EVE must report it as ZInfoAppInstMetaData. Also covers
    the appCustomStatus and hostname endpoints.
  • TestAppUserData (userdata.txt) — plain key=value user-data must reach
    the container as environment variables; #cloud-config write_files must be
    applied once per user-data version and survive an app restart.
  • TestAppLogs (app_logs.txt) — app stdout must reach the controller, both
    at container creation and continuously, and resume after stop/start.

evetest/tests/telemetry/ (TestTelemetrySuite) — from tests/lim/testdata/:

  • TestDeviceInfo (info_test.txt) — ZInfoDevice reports the applied port
    config, a plausible hardware inventory, and the HSM state.
  • TestDeviceMetrics (metric_test.txt) — DeviceMetric reports moving
    per-port network counters plus memory/CPU/controller-connectivity counters.
  • TestDeviceLogs (log_test.txt) — a message emitted on the device reaches
    the controller, and the microservice log stream is flowing.

evetest/testapps/logger-ctr/ — minimal container printing a startup banner
and numbered heartbeats. Needed because evetest-ubuntu-ctr prints nothing, so
the container-creation log path cannot be exercised with it.

Two deliberate deviations from the Eden originals:

  • metric_test matches dm.network.iName against eth0, but iName is the
    logical label from the controller config — it only equals eth0 because
    Eden's device model labels the port that way. This asserts iName and
    localName separately.
  • log_test drives SSH connections and waits for sshd's "Disconnected from".
    That trigger was tried and does not work here: sessions are established but no
    matching entry is ever uploaded. The test instead emits a per-run marker to
    /dev/kmsg — the path EVE's own ssh-service.sh uses — making the trigger
    deterministic and independent of third-party log wording.

PR dependencies

TestAppLogs pulls lfedge/evetest-logger-ctr:1.0, which is published by
publish-evetest.yml on push to master. Until that runs, the test needs the
image pushed to a reachable registry (see below).

How to test and validate this PR

All six pass on amd64/KVM against EVE 17.0.0-lts:

EVETEST_EVE_VERSION=17.0.0-lts make evetest NAME=TestTelemetrySuite
EVETEST_EVE_VERSION=17.0.0-lts make evetest NAME=TestApplicationSuite

Or individually via make evetest NAME=<TestFunctionName>. Runtimes: 147–426 s
each.

Before evetest-logger-ctr is published, TestAppLogs needs a registry the
device can reach:

docker run -d --name reg -p 5555:5000 registry:2
docker tag lfedge/evetest-logger-ctr:1.0 localhost:5555/lfedge/evetest-logger-ctr:1.0
docker push localhost:5555/lfedge/evetest-logger-ctr:1.0
EVETEST_REGISTRY_MIRROR_DOCKER=http://<host-ip>:5555 \
  EVETEST_EVE_VERSION=17.0.0-lts make evetest NAME=TestAppLogs

Changelog notes

No user-facing changes.

PR Backports

  • 17.0-stable: No, test-framework only.
  • 16.0-stable: No, test-framework only.
  • 14.5-stable: No, test-framework only.
  • 13.4-stable: No, test-framework only.

Checklist

  • I've provided a proper description
  • I've added the proper documentation — each test carries the doc comment
    structure required by evetest/README.md; the new test app is picked up
    by testapps/README.md's documented convention with no changes needed
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device — not available; nothing here is
    arch-specific, and the test app is built for both platforms by CI
  • I've written the test verification instructions
  • I've set the proper labels to this PR

@rene
rene requested a review from milan-zededa August 6, 2026 13:47
@rene
rene requested a review from eriknordmark as a code owner August 6, 2026 13:47
@rene
rene marked this pull request as draft August 6, 2026 13:47
@rene

rene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@claude

@claude

claude Bot commented Aug 6, 2026

Copy link
Copy Markdown

Claude finished @rene's task in 3m 42s —— View job


Code review

I read all 13 files, cross-checked the framework methods they use (WatchAppMetadata, GetAppLogs, RebootApplication, WatchLogs, SetConfigProperties, LogMsg/LogMsgMatch fields, the KernelLogLevel config keys), and verified the new test app follows the testapps/ convention (testapps/Makefile iterates every subdir, so logger-ctr is picked up by build-test-apps and publish-evetest.yml automatically).

This is clean, well-documented, test-only code. Framework usage is correct, the doc comments follow the README.md structure, and the two deliberate deviations from the Eden originals are sound:

  • metric_test asserting iName == "ethernet0" (logical label) and localName == "eth0" (Linux ifname) separately is more correct than Eden's eth0 match, which only worked because of Eden's device-model labelling.
  • log_test driving /dev/kmsg instead of sshd's "Disconnected from" is a reasonable, deterministic trigger over an EVE-documented ingestion path.

No correctness, security, or performance issues found. I left two low-priority inline nits on applogs_test.go:

  • Read PR diff
  • Review test packages (apps, telemetry)
  • Review new test app (logger-ctr)
  • Post inline comments
  1. Image name/tag referenced by inline literals rather than centralized constants like the ubuntu image.
  2. TestAppLogs cleanup doesn't wait for ZSwState_INVALID the way TestAppUserData does (harmless, since it's the last action under ResetDeviceConfig).

Both are optional. Note: I couldn't run go build/go vet/the tests here (the sandbox denied the build), so this is a static read only — but the author reports all six pass on amd64/KVM against 17.0.0-lts.

Comment thread evetest/tests/apps/applogs_test.go Outdated
Comment thread evetest/tests/apps/applogs_test.go Outdated
@rene

rene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

evetest-logger-ctr was created: https://hub.docker.com/r/lfedge/evetest-logger-ctr

Comment thread evetest/tests/apps/testsuite_test.go Outdated
Comment thread evetest/tests/apps/applogs_test.go Outdated
Comment thread evetest/tests/apps/applogs_test.go Outdated
Comment thread evetest/tests/apps/metadata_test.go
rene and others added 3 commits August 6, 2026 16:50
A minimal container that prints a startup banner followed by numbered
heartbeats. Log-related tests need an application that emits output from
container creation onwards; evetest-ubuntu-ctr prints nothing, so there
is no way to exercise the container-creation log path with it.

The heartbeat counter restarts from 1 on every (re)creation of the
container, which lets a test tell application restarts apart by counting
banner occurrences instead of relying on log timestamps.

Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add tests/apps, a new package for application-lifecycle scenarios that
are not about networking, holding evetest equivalents of three eden
smoke tests (tests/eclient/testdata/{metadata,userdata,app_logs}.txt):

- TestAppInstanceMetadata: an app POSTs a payload to the link-local
  metadata server and EVE must report it to the controller as
  ZInfoAppInstMetaData. Also covers the appCustomStatus endpoint, to
  check that the reported metadata type follows the endpoint used, and
  the hostname endpoint.

- TestAppUserData: ~90 KB of plain key=value user-data must not break
  deployment and must reach the container as environment variables;
  a #cloud-config document must have its write_files section applied,
  exactly once per user-data version, so that a file modified by the
  application survives an application restart.

- TestAppLogs: application stdout must reach the controller, both the
  output produced at container creation and the output produced while
  the app runs, and collection must resume after the app is stopped and
  started again.

All three are registered in TestApplicationSuite and share the same
device and network requirements so the framework reuses a single VM
across the suite.

Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add tests/telemetry, a new package covering the three channels through
which EVE reports itself to the controller, holding evetest equivalents
of three eden smoke tests (tests/lim/testdata/{log,info,metric}_test.txt):

- TestDeviceInfo: ZInfoDevice must report the port configuration EVE
  actually applied, a plausible hardware inventory, and an HSM state
  matching how the device was created.

- TestDeviceMetrics: DeviceMetric must report moving per-port network
  counters, plus the memory, CPU and controller-connectivity counters
  carried in the same message. Note that NetworkMetric.iName is the
  logical label from the controller config, not the interface name;
  eden matched it against eth0 only because its device model labels the
  port that way, so iName and localName are asserted separately here.

- TestDeviceLogs: a message emitted on the device must reach the
  controller, and the microservice log stream must be flowing. Eden
  drives SSH connections and waits for sshd's "Disconnected from"
  message; that trigger was tried here and does not work -- the sessions
  are established but no matching entry is ever uploaded. This test
  instead emits a per-run marker to /dev/kmsg, the path EVE's own
  ssh-service.sh uses with the comment "this is picked up by newlogd",
  which makes the trigger deterministic and independent of third-party
  log wording. The kernel log path is gated by debug.kernel.*.loglevel,
  separate from the debug.default.* levels the framework sets, so the
  test states those explicitly.

Fields that EVE populates conditionally are waited for inside the
Eventually predicate rather than asserted on whichever message arrived
first: DevicePort.Err is never nil (zedagent encodes port test results
into it unconditionally, so "no error" means an empty description), and
DeviceMemory/CpuMetric.TotalNs only appear once domainmgr has published
the host DomainMetric.

All three are registered in TestTelemetrySuite and state identical
device requirements so the framework reuses a single VM across the
suite.

Signed-off-by: Renê de Souza Pinto <rene@renesp.com.br>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rene
rene force-pushed the evetest-app-tests branch from 568ffb6 to 93794fc Compare August 6, 2026 15:10
@rene
rene marked this pull request as ready for review August 6, 2026 15:23

@milan-zededa milan-zededa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comments were addressed

@rene

rene commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto master (#6244) and addressed all review comments. All six tests pass on amd64/KVM against EVE 17.0.0-lts:

Test Result
TestDeviceInfo PASS (187s)
TestDeviceMetrics PASS (147s)
TestDeviceLogs PASS (173s)
TestAppInstanceMetadata PASS (201s)
TestAppUserData PASS (405s)
TestAppLogs PASS (306s)

The telemetry three were verified before the rebase (only their comments changed since); the app three were re-run after it.

Two caveats:

  1. The kubevirt path is unverified — KVM only on my side.
  2. TestAppLogs still needs lfedge/evetest-logger-ctr published. The Docker Hub repository exists but currently has zero tags (docker manifest inspect lfedge/evetest-logger-ctr:1.0no such manifest), since publish-evetest.yml pushes on merge to master. I verified the test by pushing the image into a local registry:2 and pointing EVETEST_REGISTRY_MIRROR_DOCKER at it.

Note for anyone running evetest against current master: the VERSION bump to 1.1 means lfedge/evetest-sdn:1.1 is required and it is not on Docker Hub yet — build it locally with LINUXKIT=$(realpath build-tools/bin/linuxkit) make -C evetest build-sdn-container until CI publishes it.

@milan-zededa
milan-zededa merged commit 66c150c into lf-edge:master Aug 6, 2026
41 of 62 checks passed
@rene
rene deleted the evetest-app-tests branch August 19, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants